{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification Using a Convolutional Neural Network on MXNet\n",
    "\n",
    "This tutorial is based off of Yoon Kim's [paper](https://arxiv.org/abs/1408.5882) on using convolutional neural networks for scentence sentiment classification.\n",
    "\n",
    "For this tutorial we will train a convolutional deep network model on Rotten Tomatoes movie review sentences labled with their sentiment. The result will be a model that can classify a sentence based on it's sentiment (with 1 being a purely positive sentiment, 0 being a purely negative sentiment and 0.5 being neutral).\n",
    "\n",
    "Our first step will be to fetch the labeled training data of positive and negative senitment sentences and process it into sets of vectors that are then randomly split into train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train/Dev split: 9662/1000\n",
      "train shape: (9662, 56)\n",
      "dev shape: (1000, 56)\n",
      "vocab_size 18766\n",
      "sentence max words 56\n"
     ]
    }
   ],
   "source": [
    "import urllib2\n",
    "import numpy as np\n",
    "import re\n",
    "import itertools\n",
    "from collections import Counter\n",
    "\n",
    "def clean_str(string):\n",
    "    \"\"\"\n",
    "    Tokenization/string cleaning for all datasets except for SST.\n",
    "    Original taken from https://github.com/yoonkim/CNN_sentence/blob/master/process_data.py\n",
    "    \"\"\"\n",
    "    string = re.sub(r\"[^A-Za-z0-9(),!?\\'\\`]\", \" \", string)\n",
    "    string = re.sub(r\"\\'s\", \" \\'s\", string)\n",
    "    string = re.sub(r\"\\'ve\", \" \\'ve\", string)\n",
    "    string = re.sub(r\"n\\'t\", \" n\\'t\", string)\n",
    "    string = re.sub(r\"\\'re\", \" \\'re\", string)\n",
    "    string = re.sub(r\"\\'d\", \" \\'d\", string)\n",
    "    string = re.sub(r\"\\'ll\", \" \\'ll\", string)\n",
    "    string = re.sub(r\",\", \" , \", string)\n",
    "    string = re.sub(r\"!\", \" ! \", string)\n",
    "    string = re.sub(r\"\\(\", \" \\( \", string)\n",
    "    string = re.sub(r\"\\)\", \" \\) \", string)\n",
    "    string = re.sub(r\"\\?\", \" \\? \", string)\n",
    "    string = re.sub(r\"\\s{2,}\", \" \", string)\n",
    "    return string.strip().lower()\n",
    "\n",
    "def load_data_and_labels():\n",
    "    \"\"\"\n",
    "    Loads MR polarity data from files, splits the data into words and generates labels.\n",
    "    Returns split sentences and labels.\n",
    "    \"\"\"\n",
    "    # Pull scentences with positive sentiment\n",
    "    pos_file = urllib2.urlopen('https://raw.githubusercontent.com/yoonkim/CNN_sentence/master/rt-polarity.pos')\n",
    "\n",
    "    # Pull scentences with negative sentiment\n",
    "    neg_file = urllib2.urlopen('https://raw.githubusercontent.com/yoonkim/CNN_sentence/master/rt-polarity.neg')\n",
    "    \n",
    "    # Load data from files\n",
    "    positive_examples = list(pos_file.readlines())\n",
    "    positive_examples = [s.strip() for s in positive_examples]\n",
    "    negative_examples = list(neg_file.readlines())\n",
    "    negative_examples = [s.strip() for s in negative_examples]\n",
    "    # Split by words\n",
    "    x_text = positive_examples + negative_examples\n",
    "    x_text = [clean_str(sent) for sent in x_text]\n",
    "    x_text = [s.split(\" \") for s in x_text]\n",
    "    # Generate labels\n",
    "    positive_labels = [1 for _ in positive_examples]\n",
    "    negative_labels = [0 for _ in negative_examples]\n",
    "    y = np.concatenate([positive_labels, negative_labels], 0)\n",
    "    return [x_text, y]\n",
    "\n",
    "\n",
    "def pad_sentences(sentences, padding_word=\"</s>\"):\n",
    "    \"\"\"\n",
    "    Pads all sentences to the same length. The length is defined by the longest sentence.\n",
    "    Returns padded sentences.\n",
    "    \"\"\"\n",
    "    sequence_length = max(len(x) for x in sentences)\n",
    "    padded_sentences = []\n",
    "    for i in range(len(sentences)):\n",
    "        sentence = sentences[i]\n",
    "        num_padding = sequence_length - len(sentence)\n",
    "        new_sentence = sentence + [padding_word] * num_padding\n",
    "        padded_sentences.append(new_sentence)\n",
    "    return padded_sentences\n",
    "\n",
    "\n",
    "def build_vocab(sentences):\n",
    "    \"\"\"\n",
    "    Builds a vocabulary mapping from word to index based on the sentences.\n",
    "    Returns vocabulary mapping and inverse vocabulary mapping.\n",
    "    \"\"\"\n",
    "    # Build vocabulary\n",
    "    word_counts = Counter(itertools.chain(*sentences))\n",
    "    # Mapping from index to word\n",
    "    vocabulary_inv = [x[0] for x in word_counts.most_common()]\n",
    "    # Mapping from word to index\n",
    "    vocabulary = {x: i for i, x in enumerate(vocabulary_inv)}\n",
    "    return [vocabulary, vocabulary_inv]\n",
    "\n",
    "\n",
    "def build_input_data(sentences, labels, vocabulary):\n",
    "    \"\"\"\n",
    "    Maps sentencs and labels to vectors based on a vocabulary.\n",
    "    \"\"\"\n",
    "    x = np.array([[vocabulary[word] for word in sentence] for sentence in sentences])\n",
    "    y = np.array(labels)\n",
    "    return [x, y]\n",
    "\n",
    "\"\"\"\n",
    "Loads and preprocessed data for the MR dataset.\n",
    "Returns input vectors, labels, vocabulary, and inverse vocabulary.\n",
    "\"\"\"\n",
    "# Load and preprocess data\n",
    "sentences, labels = load_data_and_labels()\n",
    "sentences_padded = pad_sentences(sentences)\n",
    "vocabulary, vocabulary_inv = build_vocab(sentences_padded)\n",
    "x, y = build_input_data(sentences_padded, labels, vocabulary)\n",
    "\n",
    "vocab_size = len(vocabulary)\n",
    "\n",
    "# randomly shuffle data\n",
    "np.random.seed(10)\n",
    "shuffle_indices = np.random.permutation(np.arange(len(y)))\n",
    "x_shuffled = x[shuffle_indices]\n",
    "y_shuffled = y[shuffle_indices]\n",
    "\n",
    "# split train/dev set\n",
    "# there are a total of 10662 labled examples to train on\n",
    "x_train, x_dev = x_shuffled[:-1000], x_shuffled[-1000:]\n",
    "y_train, y_dev = y_shuffled[:-1000], y_shuffled[-1000:]\n",
    "\n",
    "sentence_size = x_train.shape[1]\n",
    "\n",
    "print 'Train/Dev split: %d/%d' % (len(y_train), len(y_dev))\n",
    "print 'train shape:', x_train.shape\n",
    "print 'dev shape:', x_dev.shape\n",
    "print 'vocab_size', vocab_size\n",
    "print 'sentence max words', sentence_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we prepared the training and test data by loading, vectorizing and shuffling it we can go on to defining the network architecture we want to train with the data.\n",
    "\n",
    "We will first set up some placeholders for the input and output of the network then define the first layer, an embedding layer, which learns to map word vectors into a lower dimensional vector space where distances between words correspond to how related they are (with respect to sentiment they convey)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch size 50\n",
      "embedding dimensions 300\n"
     ]
    }
   ],
   "source": [
    "import mxnet as mx\n",
    "import sys,os\n",
    "\n",
    "'''\n",
    "Define batch size and the place holders for network inputs and outputs\n",
    "'''\n",
    "\n",
    "batch_size = 50 # the size of batches to train network with\n",
    "print 'batch size', batch_size\n",
    "\n",
    "input_x = mx.sym.Variable('data') # placeholder for input data\n",
    "input_y = mx.sym.Variable('softmax_label') # placeholder for output label\n",
    "\n",
    "\n",
    "'''\n",
    "Define the first network layer (embedding)\n",
    "'''\n",
    "\n",
    "# create embedding layer to learn representation of words in a lower dimensional subspace (much like word2vec)\n",
    "num_embed = 300 # dimensions to embed words into\n",
    "print 'embedding dimensions', num_embed\n",
    "\n",
    "embed_layer = mx.sym.Embedding(data=input_x, input_dim=vocab_size, output_dim=num_embed, name='vocab_embed')\n",
    "\n",
    "# reshape embedded data for next layer\n",
    "conv_input = mx.sym.Reshape(data=embed_layer, target_shape=(batch_size, 1, sentence_size, num_embed))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next layer in the network performs convolutions over the ordered embedded word vectors in a sentence using multiple filter sizes, sliding over 3, 4 or 5 words at a time. This is the equivalent of looking at all 3-grams, 4-grams and 5-grams in a sentence and will allow us to understand how words contribute to sentiment in the context of those around them.\n",
    "\n",
    "After each convolution we add a max-pool layer to extract the most significant elements in each convolution and turn them into a feature vector.\n",
    "\n",
    "Because each convolution+pool filter produces tensors of different shapes we need to create a layer for each of them, and then concatonate the results of these layers into one big feature vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "convolution filters [3, 4, 5]\n"
     ]
    }
   ],
   "source": [
    "# create convolution + (max) pooling layer for each filter operation\n",
    "filter_list=[3, 4, 5] # the size of filters to use\n",
    "print 'convolution filters', filter_list\n",
    "\n",
    "num_filter=100\n",
    "pooled_outputs = []\n",
    "for i, filter_size in enumerate(filter_list):\n",
    "    convi = mx.sym.Convolution(data=conv_input, kernel=(filter_size, num_embed), num_filter=num_filter)\n",
    "    relui = mx.sym.Activation(data=convi, act_type='relu')\n",
    "    pooli = mx.sym.Pooling(data=relui, pool_type='max', kernel=(sentence_size - filter_size + 1, 1), stride=(1,1))\n",
    "    pooled_outputs.append(pooli)\n",
    "\n",
    "# combine all pooled outputs\n",
    "total_filters = num_filter * len(filter_list)\n",
    "concat = mx.sym.Concat(*pooled_outputs, dim=1)\n",
    "\n",
    "# reshape for next layer\n",
    "h_pool = mx.sym.Reshape(data=concat, target_shape=(batch_size, total_filters))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we add dropout regularization, which will randomly disable a fraction of neruons in the layer (set to 50% here) to ensure that that model does not overfit. This works by preventing neurons from co-adapting and forcing them to learn individually useful features. \n",
    "\n",
    "This is nessecary in our model becasuse the dataset has a vocabulary of size around 20k and only around 10k examples so since this data set is pretty small we’re likely to overfit with a powerful model (like this neural net)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dropout probablity 0.5\n"
     ]
    }
   ],
   "source": [
    "# dropout layer\n",
    "dropout=0.5\n",
    "print 'dropout probablity', dropout\n",
    "\n",
    "if dropout > 0.0:\n",
    "    h_drop = mx.sym.Dropout(data=h_pool, p=dropout)\n",
    "else:\n",
    "    h_drop = h_pool\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we add a fully connected layer to add non-linearity to the model. We then classify the resulting output of this layer using a softmax function, yeilding a result between 0 (negative sentimet) and 1 (positive)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# fully connected layer\n",
    "num_label=2\n",
    "\n",
    "cls_weight = mx.sym.Variable('cls_weight')\n",
    "cls_bias = mx.sym.Variable('cls_bias')\n",
    "\n",
    "fc = mx.sym.FullyConnected(data=h_drop, weight=cls_weight, bias=cls_bias, num_hidden=num_label)\n",
    "\n",
    "# softmax output\n",
    "sm = mx.sym.SoftmaxOutput(data=fc, label=input_y, name='softmax')\n",
    "\n",
    "# set CNN pointer to the \"back\" of the network\n",
    "cnn = sm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have defined our CNN model we will define the device on our machine that we will train and execute this model on, as well as the datasets to train and test this model with.\n",
    "\n",
    "*If you are running this code be sure that you have a GPU on your machine if your ctx is set to mx.gpu(0) otherwise you can set your ctx to mx.cpu(0) which will run the training much slower*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "import time\n",
    "import math\n",
    "\n",
    "# Define the structure of our CNN Model (as a named tuple)\n",
    "CNNModel = namedtuple(\"CNNModel\", ['cnn_exec', 'symbol', 'data', 'label', 'param_blocks'])\n",
    "\n",
    "# Define what device to train/test on\n",
    "ctx=mx.gpu(0)\n",
    "# If you have no GPU on your machine change this to \n",
    "# ctx=mx.cpu(0)\n",
    "\n",
    "arg_names = cnn.list_arguments()\n",
    "\n",
    "input_shapes = {}\n",
    "input_shapes['data'] = (batch_size, sentence_size)\n",
    "\n",
    "arg_shape, out_shape, aux_shape = cnn.infer_shape(**input_shapes)\n",
    "arg_arrays = [mx.nd.zeros(s, ctx) for s in arg_shape]\n",
    "args_grad = {}\n",
    "for shape, name in zip(arg_shape, arg_names):\n",
    "    if name in ['softmax_label', 'data']: # input, output\n",
    "        continue\n",
    "    args_grad[name] = mx.nd.zeros(shape, ctx)\n",
    "\n",
    "cnn_exec = cnn.bind(ctx=ctx, args=arg_arrays, args_grad=args_grad, grad_req='add')\n",
    "\n",
    "param_blocks = []\n",
    "arg_dict = dict(zip(arg_names, cnn_exec.arg_arrays))\n",
    "initializer=mx.initializer.Uniform(0.1)\n",
    "for i, name in enumerate(arg_names):\n",
    "    if name in ['softmax_label', 'data']: # input, output\n",
    "        continue\n",
    "    initializer(name, arg_dict[name])\n",
    "\n",
    "    param_blocks.append( (i, arg_dict[name], args_grad[name], name) )\n",
    "\n",
    "out_dict = dict(zip(cnn.list_outputs(), cnn_exec.outputs))\n",
    "\n",
    "data = cnn_exec.arg_dict['data']\n",
    "label = cnn_exec.arg_dict['softmax_label']\n",
    "\n",
    "cnn_model= CNNModel(cnn_exec=cnn_exec, symbol=cnn, data=data, label=label, param_blocks=param_blocks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now execute the training and testing of our network, which in-part mxnet automatically does for us with its forward and backwards propogation methods, along with its automatic gradient calculations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "'''\n",
    "Train the cnn_model using back prop\n",
    "''' \n",
    "\n",
    "optimizer='rmsprop'\n",
    "max_grad_norm=5.0\n",
    "learning_rate=0.0005\n",
    "epoch=50\n",
    "\n",
    "print 'optimizer', optimizer\n",
    "print 'maximum gradient', max_grad_norm\n",
    "print 'learning rate (step size)', learning_rate\n",
    "print 'epochs to train for', epoch\n",
    "\n",
    "# create optimizer\n",
    "opt = mx.optimizer.create(optimizer)\n",
    "opt.lr = learning_rate\n",
    "\n",
    "updater = mx.optimizer.get_updater(opt)\n",
    "\n",
    "# create logging output\n",
    "logs = sys.stderr\n",
    "\n",
    "# For each training epoch\n",
    "for iteration in range(epoch):\n",
    "    tic = time.time()\n",
    "    num_correct = 0\n",
    "    num_total = 0\n",
    "    \n",
    "    # Over each batch of training data\n",
    "    for begin in range(0, x_train.shape[0], batch_size):\n",
    "        batchX = x_train[begin:begin+batch_size]\n",
    "        batchY = y_train[begin:begin+batch_size]\n",
    "        if batchX.shape[0] != batch_size:\n",
    "            continue\n",
    "\n",
    "        cnn_model.data[:] = batchX\n",
    "        cnn_model.label[:] = batchY\n",
    "\n",
    "        # forward\n",
    "        cnn_model.cnn_exec.forward(is_train=True)\n",
    "\n",
    "        # backward\n",
    "        cnn_model.cnn_exec.backward()\n",
    "\n",
    "        # eval on training data\n",
    "        num_correct += sum(batchY == np.argmax(cnn_model.cnn_exec.outputs[0].asnumpy(), axis=1))\n",
    "        num_total += len(batchY)\n",
    "\n",
    "        # update weights\n",
    "        norm = 0\n",
    "        for idx, weight, grad, name in cnn_model.param_blocks:\n",
    "            grad /= batch_size\n",
    "            l2_norm = mx.nd.norm(grad).asscalar()\n",
    "            norm += l2_norm * l2_norm\n",
    "\n",
    "        norm = math.sqrt(norm)\n",
    "        for idx, weight, grad, name in cnn_model.param_blocks:\n",
    "            if norm > max_grad_norm:\n",
    "                grad *= (max_grad_norm / norm)\n",
    "\n",
    "            updater(idx, grad, weight)\n",
    "\n",
    "            # reset gradient to zero\n",
    "            grad[:] = 0.0\n",
    "\n",
    "    # Decay learning rate for this epoch to ensure we are not \"overshooting\" optima\n",
    "    if iteration % 50 == 0 and iteration > 0:\n",
    "        opt.lr *= 0.5\n",
    "        print >> logs, 'reset learning rate to %g' % opt.lr\n",
    "\n",
    "    # End of training loop for this epoch\n",
    "    toc = time.time()\n",
    "    train_time = toc - tic\n",
    "    train_acc = num_correct * 100 / float(num_total)\n",
    "\n",
    "    # Saving checkpoint to disk\n",
    "    if (iteration + 1) % 10 == 0:\n",
    "        prefix = 'cnn'\n",
    "        cnn_model.symbol.save('./%s-symbol.json' % prefix)\n",
    "        save_dict = {('arg:%s' % k) :v  for k, v in cnn_model.cnn_exec.arg_dict.items()}\n",
    "        save_dict.update({('aux:%s' % k) : v for k, v in cnn_model.cnn_exec.aux_dict.items()})\n",
    "        param_name = './%s-%04d.params' % (prefix, iteration)\n",
    "        mx.nd.save(param_name, save_dict)\n",
    "        print >> logs, 'Saved checkpoint to %s' % param_name\n",
    "\n",
    "\n",
    "    # Evaluate model after this epoch on dev (test) set\n",
    "    num_correct = 0\n",
    "    num_total = 0\n",
    "    \n",
    "    # For each test batch\n",
    "    for begin in range(0, x_dev.shape[0], batch_size):\n",
    "        batchX = x_dev[begin:begin+batch_size]\n",
    "        batchY = y_dev[begin:begin+batch_size]\n",
    "\n",
    "        if batchX.shape[0] != batch_size:\n",
    "            continue\n",
    "\n",
    "        cnn_model.data[:] = batchX\n",
    "        cnn_model.cnn_exec.forward(is_train=False)\n",
    "\n",
    "        num_correct += sum(batchY == np.argmax(cnn_model.cnn_exec.outputs[0].asnumpy(), axis=1))\n",
    "        num_total += len(batchY)\n",
    "\n",
    "    dev_acc = num_correct * 100 / float(num_total)\n",
    "    print >> logs, 'Iter [%d] Train: Time: %.3fs, Training Accuracy: %.3f \\\n",
    "            --- Dev Accuracy thus far: %.3f' % (iteration, train_time, train_acc, dev_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have gone through the trouble of training the model, we have stored the learned parameters in the .params file in our local directory. We can now load this file whenever we want and predict the sentiment of new sentences by running them through a forward pass of the trained model.\n",
    "\n",
    "## References\n",
    "- [“Implementing a CNN for Text Classification in Tensorflow” blog post](http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/)\n",
    "- [Convolutional Neural Networks for Sentence Classification](https://arxiv.org/abs/1408.5882)\n"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
